Papers with information mixing
Quantifying Context Mixing in Transformers (2023.eacl-main)
Copied to clipboard
| Challenge: | Self-attention weights and their transformed variants have been used for analyzing token-to-token interactions in Transformer-based models, but they are not faithful to the models’ decisions as they are only one part of an encoder block. |
| Approach: | They propose a new context mixing score customized for Transformers that provides us with a deeper understanding of how information is mixed at each encoder layer. |
| Outcome: | The proposed score outperforms other methods in linguistically informed rationales, probing, and faithfulness analysis. |
One Token Is Enough: Improving Diffusion Language Models with a Sink Token (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing Diffusion Language Models lack a structural constraint to stabilize attention sinks. |
| Approach: | They propose a simple but effective extra sink token that is constrained to attend to itself while remaining globally visible to all other tokens. |
| Outcome: | The proposed token is able to stabilize attention sinks and improve model performance. |